1. Confusion Matrix

Example 1

confusion_matrix.PNG

Example 2

confusion_matrix2.PNG

Type 1 and Type 2 Errors

  1. Type 1 Error (Error of the first kind, or False Positive): In the medical example, this is when we misdiagnose a healthy patient as sick.
  2. Type 2 Error (Error of the second kind, or False Negative): In the medical example, this is when we misdiagnose a sick patient as healthy.

Understanding the Confusion Matrix

  1. True Positive (TP) is the number of correct predictions that an example is positive. This means positive class correctly identified as positive. Example: Given class is spam and the classifier has been correctly predicted it as spam.
  2. False Negative (FN) is the number of incorrect predictions that an example is negative. This means positive class incorrectly identified as negative. Example: Given class is spam however, the classifier has been incorrectly predicted it as non-spam.
  3. False Negative (FN) is the number of incorrect predictions that an example is negative. This means positive class incorrectly identified as negative. Example: Given class is spam however, the classifier has been incorrectly predicted it as non-spam.
  4. False Negative (FN) is the number of incorrect predictions that an example is negative. This means positive class incorrectly identified as negative. Example: Given class is spam however, the classifier has been incorrectly predicted it as non-spam.

2. Accuracy

accuracy.PNG

accuracy2.PNG

Question: When Accuracy don't work?

Example: Model 1: All transactions are good

problem_w_accuracy.PNG

Example: Model 2: All transactions are fraudulent

problem_w_accuracy2.PNG

3. False Negatives & False Positives

Question: In the medical example, what is worse, a False Positive, or a False Negative?
Answer: A False Positive implies sending a healthy person to get more tests. This is slightly inconvenient, but ok. A False Negative implies sending a sick person home, which can be disastrous!

medical.png

Question: In the email example, what is worse, a False Positive, or a False Negative?
Answer: A False Negative implies a spam message will make its way into your inbox. This is slightly inconvenient, but ok. A False Positive implies missing an e-mail from your dear grandma, which can be disastrous!

email.png

4. Precision and Recall

precision_recall.PNG

confusionMatrxiUpdated.jpg

A. Precision

precision_1.PNG

precision2.PNG

B. Recall

recall1.PNG

recall2.PNG

5. F1 Score

-F1 score is a weighted average of the recall (sensitivity) and precision. F1 score might be good choice when you seek to balance between Precision and Recall.

f%20score.jpg

f1_score.PNG

Question: What averages of recall and precision is not a good idea, but weighted average it?

Example: Model 1: All transactions are good

f1_1.PNG

Example: Model 2: All transactions are fraudulent

f1_2.PNG

F1 score = Harmonic Mean of Precision and Recall

hm.PNG

Example: Model 1: All transactions are good

hm_1.PNG

6. F-beta Score

F_beta_formula.PNG

Understanding F-beta score

F_beta.PNG

Question: Out of the following three models, which one should have an F-beta score of 2, 1, and 0.5? Match each model with its corresponding score.

  1. Detecting malfunctioning parts in a spaceship
  1. Sending phone notifications about videos a user may like
  1. Sending promotional material in the mail to potential clients

Boundaries_of_F_beta_score.PNG

7. Specificity

8. Receiver Operating Characteristic (ROC) Curve

Roc1.PNG

Example:

roc2.PNG

roc3.PNG

Question: What happens when the split is Perfect?

Question: What happens when the split is Random (Bad)?

Summarizing

roc_summary.PNG

Conclusion:

Question: Can the area under the ROC curve be less than 0.5?

9. Regression Metrics

A. Mean Absolute Error

MAE_1.PNG

B. Mean Squared Error

MSE.PNG

C. RMSE

D. R- Squared

R_squared.png

R2_eg.PNG

R2_python.PNG

E. Adjusted R-Squared

Adj_r2_formula.PNG